Changing view order of augmented reality objects based on user gaze

ABSTRACT

Disclosed embodiments provide techniques for adjusting the Z-index of an augmented reality (AR) object. When an AR object covers a first object, the user gazes at the first object for a predetermined amount of time, and then the AR object Z-index is adjusted with respect to the Z-index of the first object, such that the AR object is now rendered behind the first object, such that the user can continue to view the first object. The first object can be a physical object, a virtual object, or another augmented reality object. This allows the user to conveniently continue viewing the first object. Embodiments provide techniques for reverting the Z-index of the AR object at a later time when the user is ready to respond to the AR object. In this way, the user experience for augmented reality systems is enhanced.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present patent document is a continuation of U.S. patent applicationSer. No. 15/819,571, filed Nov. 21, 2017, entitled “CHANGING VIEW ORDEROF AUGMENTED REALITY OBJECTS BASED ON USER GAZE”, the entire contents ofwhich is incorporated herein by reference.

FIELD OF THE INVENTION

Embodiments relate to augmented reality, and more particularly tochanging the view order of augmented reality objects based on user gaze.

BACKGROUND

Augmented reality allows people to view their surroundings in the realworld, usually through a set of eyeglasses and/or a screen, supplementedwith virtual (i.e., augmented reality) images. This allows for anenhanced experience for the person exploring his/her surroundings. Insome cases, though, a virtual image may obscure a real object or anothervirtual image that the user is interested in looking at. This may causeconfusion and frustration for the user, as well as reduces efficiency.Accordingly, there exists a need for improvements in augmented realitysystems and methods.

SUMMARY

In one aspect, there is provided a computer-implemented method forcontrolling a Z-index of an augmented reality object in an augmentedreality system, the method comprising: detecting a first object ascurrently focused on by a user; detecting the presence of an overlappingaugmented reality object that obstructs a view of the first object; and,responsive to the user maintaining gaze on the first object for at leasta predetermined period of time, adjusting the Z-index for the augmentedreality object.

In another aspect, there is provided an electronic communication devicecomprising: a forward-facing digital camera; a user-facing digitalcamera; a processor; a memory coupled to the processor, the memorycontaining instructions, that when executed by the processor, performthe steps of: detecting a first object as currently focused on by auser; detecting the presence of an overlapping augmented reality objectthat obstructs a view of the first object; and, responsive to the usermaintaining gaze on the first object for at least a predetermined periodof time, adjusting a Z-index for the augmented reality object.

In yet another aspect, there is provided a computer program product forcontrolling a Z-index of an augmented reality object in an augmentedreality system, for an electronic computing device comprising a computerreadable storage medium having program instructions embodied therewith,the program instructions executable by a processor to cause theelectronic computing device to: detect a first object as currentlyfocused on by a user; detect the presence of an overlapping augmentedreality object that obstructs a view of the first object; and,responsive to the user maintaining gaze on the first object for at leasta predetermined period of time, adjusting a Z-index for the augmentedreality object.

BRIEF DESCRIPTION OF THE DRAWINGS

Features of the disclosed embodiments will be more readily understoodfrom the following detailed description of the various aspects of theinvention taken in conjunction with the accompanying drawings.

FIG. 1 is a block diagram of a device in accordance with embodiments ofthe present invention.

FIG. 2 is an exemplary device in accordance with embodiments of thepresent invention.

FIG. 3 is an exemplary display of an augmented reality object renderedon a device in accordance with embodiments of the present invention.

FIG. 4A is an exemplary display of an augmented reality object with anadjusted Z-index rendered on a device in accordance with embodiments ofthe present invention.

FIG. 4B is an exemplary display of an augmented reality object with anadjusted Z-index rendered on a device in accordance with embodiments ofthe present invention.

FIG. 4C is an exemplary display of an augmented reality object with aconfirmation message rendered on a device in accordance with embodimentsof the present invention.

FIG. 4D is an exemplary display of an augmented reality object moved toa non-overlapping location with respect to a first object.

FIG. 5 illustrates a side view of a usage of embodiments of the presentinvention with an augmented reality object in front of a physicalobject.

FIG. 6 illustrates a side view of a usage of embodiments of the presentinvention with an augmented reality object behind a physical object.

FIG. 7 shows an eyewear apparatus in accordance with embodiments of thepresent invention.

FIG. 8 is a flowchart 800 indicating process steps for embodiments ofthe present invention.

FIG. 9 shows additional details of an embodiment of the presentinvention.

The drawings are not necessarily to scale. The drawings are merelyrepresentations, not necessarily intended to portray specific parametersof the invention. The drawings are intended to depict only exampleembodiments of the invention, and therefore should not be considered aslimiting in scope. In the drawings, like numbering may represent likeelements. Furthermore, certain elements in some of the Figures may beomitted, or illustrated not-to-scale, for illustrative clarity.

DETAILED DESCRIPTION

Disclosed embodiments provide techniques for adjusting the Z-index of anaugmented reality (AR) object. The Z-index is an indication of where anobject is displayed with respect to other objects in terms of foregroundand background. As an example, a Z-index of zero may indicate an objectthat is in front of any other object, a Z-index of one may indicate anobject that is rendered as behind objects with a Z-index of zero, and infront of any objects with a Z-index of two or higher. In someembodiments, when an AR object covers a first object, the user gazes inthe direction of the first object for a predetermined amount of time,and then the AR object Z-index is adjusted with respect to the Z-indexof the first object, such that the AR object is now rendered behind thefirst object, such that the user can continue to view the first object.The first object can be a physical object, a virtual reality object, oranother augmented reality object. This allows the user to convenientlycontinue viewing the first object. Embodiments provide techniques forreverting the Z-index of the AR object at a later time when the user isready to respond to the AR object. In this way, the user experience foraugmented reality systems is enhanced.

Reference throughout this specification to “one embodiment,” “anembodiment,” “some embodiments”, or similar language means that aparticular feature, structure, or characteristic described in connectionwith the embodiment is included in at least one embodiment of thepresent invention. Thus, appearances of the phrases “in one embodiment,”“in an embodiment,” “in some embodiments”, and similar languagethroughout this specification may, but do not necessarily, all refer tothe same embodiment.

Moreover, the described features, structures, or characteristics of theinvention may be combined in any suitable manner in one or moreembodiments. It will be apparent to those skilled in the art thatvarious modifications and variations can be made to the presentinvention without departing from the spirit and scope and purpose of theinvention. Thus, it is intended that the present invention cover themodifications and variations of this invention provided they come withinthe scope of the appended claims and their equivalents. Reference willnow be made in detail to the preferred embodiments of the invention.

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of this disclosure.As used herein, the singular forms “a”, “an”, and “the” are intended toinclude the plural forms as well, unless the context clearly indicatesotherwise. Furthermore, the use of the terms “a”, “an”, etc., do notdenote a limitation of quantity, but rather denote the presence of atleast one of the referenced items. The term “set” is intended to mean aquantity of at least one. It will be further understood that the terms“comprises” and/or “comprising”, or “includes” and/or “including”, or“has” and/or “having”, when used in this specification, specify thepresence of stated features, regions, integers, steps, operations,elements, and/or components, but do not preclude the presence oraddition of one or more other features, regions, or elements.

FIG. 1 is a block diagram of a device in accordance with embodiments ofthe present invention. Device 100 is shown as a simplified diagram ofmodules. Device 100 is an electronic computing device. Device 100includes a processor 102, which is coupled to a memory 104. Memory 104may include dynamic random access memory (DRAM), static random accessmemory (SRAM), magnetic storage, and/or a read only memory such asflash, EEPROM, optical storage, or other suitable memory. In someembodiments, memory 104 may not be a transitory signal per se. Memory104 includes instructions, which when executed by the processor,implement steps of the present invention. In embodiments, device 100 mayhave multiple processors 102, and/or multiple cores per processor.

Device 100 further includes a user interface 114. In some embodiments,the user interface may include a display system, which may include oneor more displays, examples of which include a liquid crystal display(LCD), a plasma display, a cathode ray tube (CRT) display, a lightemitting diode (LED) display, an organic LED (OLED) display, or othersuitable display technology. The user interface 114 may include akeyboard, mouse, and/or a touch screen, incorporating a capacitive orresistive touch screen in some embodiments. The device 100 furtherincludes a microphone 120. The device 100 further includes a user-facingcamera 118 and a forward-facing camera 108.

Device 100 further includes a network interface 112. In someembodiments, the network interface 112 may include a wirelesscommunication interface that includes modulators, demodulators, andantennas for a variety of wireless protocols including, but not limitedto, Bluetooth™, Wi-Fi, and/or cellular communication protocols forcommunication over a computer network. Any communication interface, nowknown or hereafter developed, may be substituted.

The device 100 may further include an accelerometer 116. Theaccelerometer may be bulk micromachined capacitive, bulk micromachinedpiezoelectric resistive, capacitive spring mass system base, DCresponse, electromechanical servo (Servo Force Balance), high gravity,high temperature, laser accelerometer, or other suitable type.

The device 100 may further include a geolocation receiver 110.Geolocation receiver 110 can operate with one or more of GPS, Galileo,GLONASS, or other system now known or hereafter developed.

FIG. 2 is an exemplary device 200 in accordance with embodiments of thepresent invention. Device 200 includes display 204 and user-facingcamera 218. This camera faces the user when the user is looking at thedisplay 204 of device 200. In the example, on the display 204 is animage of a house 206 having a door 209 and a window 211. The house 206may be a real house captured by a forward-facing camera (a camera thatfaces forward when the user is looking at display 204), or a virtualhouse.

FIG. 3 is an exemplary display of an augmented reality object renderedon device 300 in accordance with embodiments of the present invention.In embodiments, it is detected that a user's gaze is currently focusedon a first object. The detecting is implemented by analysis of imagerycaptured by the user facing camera 218. In embodiments, it is detectedthat an overlapping AR object is present that obstructs the view of thefirst object. In the example, there are two AR objects, includingcharacter 224 and text box 225. These AR objects 224 and 225 obscureportions of door 209 and window 211.

FIG. 4A is an exemplary display 400 of an augmented reality object withan adjusted Z-index rendered on a device in accordance with embodimentsof the present invention. In embodiments, responsive to the usermaintaining gaze on the first object for at least a predetermined periodof time, a new Z-index is set for the augmented reality object. TheZ-index is an indication of where an object is displayed with respect toother objects in terms of foreground and background.

In some embodiments, adjusting the Z-index includes changing an opacityparameter for the augmented reality object. In embodiments, an AR object“goes behind another object” by setting the AR object's pixels totransparent in an overlap region. In some embodiments, alpha compositingmay be used. In the example, pixels are of the RGBA model. Accordingly,the pixel color includes a red, green, and blue component, along with analpha channel. The alpha channel indicates transparency, ranging fromopaque to invisible. A pixel having a value of 0% in its alpha channelis completely transparent. This means it is invisible. A pixel having avalue of 100% in its alpha channel is completely opaque. Values may beset anywhere between 0 and 100%, allowing for no, full, or partialtransparency of pixels. Alpha of a pixel may be set to transparent whenset “behind” another object, and opaque when set in “front” of anotherobject. Thus, the transparency adjustment can be performed on apixel-by-pixel basis. It should be recognized that any type of pixelformats and compositing processes may be substituted where feasible.

In the example, the predetermined period of time, after which Z-indexadjustment is triggered, is six seconds. It is detected (by analysis ofimagery from user-facing camera 218) that, after the AR objects 224 and225 appear, the user's gaze focuses on door 209 for at least sixseconds. Accordingly, the Z-indexes for the AR objects 224 and 225 areadjusted. The alpha channels for the pixels of AR objects 224 and 225that overlap with the door 209, are set to a value of zero (i.e.,transparent). This means that the AR objects 224 and 225 are moved fromin front of, to behind, the door 209. AR object text box 225 remains infront of the window because the user eye gaze is directed at the door209.

FIG. 4B is an exemplary display 403 of an augmented reality object withan adjusted Z-index rendered on a device in accordance with additionalembodiments of the present invention. In the example, the user moveshis/her gaze to AR object 225. S/he stares at it for six seconds.Accordingly, the Z-index is adjusted for AR object 225 to cause it to bebehind the door 209, but in front of window 211, as was the case forFIG. 4A. However, in the case of FIG. 4B, the opacity parameter (alphachannel) is adjusted such that the AR object 225 is partiallytransparent in front of window 211. Accordingly, here, instead of thealpha channel of the pixels that overlap with the window being set to 0for complete transparency, they are instead set at a mid-level betweentransparent and opaque to allow partial transparency. Accordingly, thepixels of the window are still visible to an extent even though ARobject 225 partially overlaps it.

FIG. 4C is an exemplary display 405 of an augmented reality object witha confirmation message rendered on a device in accordance withembodiments of the present invention. In embodiments, a confirmationmessage is rendered prior to setting the new Z-index for the augmentedreality object. In the example, instead of automatically adjusting theZ-index for the AR object in response to detecting user gaze for theperiod of time, a confirmation message is displayed to determine whetherthe user wants the adjustment. In the example, text box 227 is displayedrequesting confirmation from the user as to whether the text box shouldbe moved back. The user can answer by a voice command, or by eye gazingon the “YES” or “NO” areas, or any other gesture or suitable user input.Thus, embodiments can include rendering a confirmation message prior tosetting the new Z-index for the augmented reality object.

FIG. 4D is an exemplary display 407 of an augmented reality object movedto a non-overlapping location with respect to a first object. Inembodiments, in addition to (or instead of) changing the Z-index, the ARobjects 224 and 225 are moved to a different part of the display wherethey no longer overlap with the door 209. Thus embodiments includemoving a position of the overlapping augmented reality object to anon-overlapping location with respect to the first object.

FIG. 5 illustrates a diagram 500 of a side view of a usage ofembodiments of the present invention with an augmented reality object infront of a physical object. Device 505 has user facing camera 518 andforward facing camera 508. The user 502 has eye 541 gazing along a lineof sight 542, that is directed towards screen 503 of device 505. Line544 is an extrapolation of the line of sight extending to door 509 ofbuilding 506. Augmented reality object 524 is shown positioned in frontof door 509. The building 506 and door 509 are real objects captured inthe forward facing camera 508, but the AR object shown is not a realobject, but instead a representation displayed on screen 503 of device505.

FIG. 6 illustrates a diagram 600 of a side view of a usage ofembodiments of the present invention with an augmented reality objectbehind a physical object. Embodiments may include, responsive todetecting an eye gesture, reverting the Z-index for the augmentedreality object. In the example, the scenario is the same as in FIG. 5,except that now the AR object 524 is reverted to a position behind thedoor 509, as a result of a detection of the user gazing at the directionof the door 509 for a predetermined amount of time (e.g., six seconds).

It should be recognized that diagrams 500 and 600 are representations.The AR object is not real and, accordingly, is not actually positionedin front of or behind the door in reality. The AR object is shown on thedisplay 503. The display also shows the real world objects captured byforward facing camera 508.

The eye gesture may be any suitable gesture. In some embodiments,detecting an eye gesture comprises detecting a blink pattern. In someembodiments, the detecting of a blink pattern comprises detecting apredetermined first time period of closed eyes, followed by apredetermined second time period of opened eyes. For example, thepattern may be three seconds of eyes closed, followed by five seconds ofeyes open. If the system detects that pattern, then the Z-index of therelevant AR object is adjusted (e.g. reverted).

In some embodiments, detecting a blink pattern comprises detecting apredetermined first time period of a first eye closed and a second eyeopened, followed by a predetermined second time period of the first eyeopened and the second eye opened. For example, the pattern may be threeseconds of a first eye opened and a second eye closed, followed by sixseconds of the first eye opened and the second eye opened. If the systemdetects that pattern, then the Z-index of the relevant AR object isadjusted.

In embodiments, detecting an eye gesture comprises detecting a winkpattern. In some embodiments, the detecting of a wink pattern comprisesdetecting a predetermined number of winks, followed by a predeterminedsecond time period with eyes open, followed by a second predeterminednumber of winks. For example, the pattern may be four winks, followed bythree seconds of eyes open, followed by three winks. If the systemdetects that pattern, then the Z-index of the relevant AR object isadjusted.

In embodiments, detecting an eye gesture comprises detecting an eye gazepattern. In some embodiments, the detecting of an eye gaze patterncomprises detecting a gaze in a particular direction for a predeterminedperiod of time, followed by a gaze in another direction for a secondpredetermined period of time. The direction could be left, right, up,down, etc. It can be instead a rolling of the eyes—from the left to theright, or top to bottom, etc. For example, the pattern may be a gaze tothe left for two seconds, followed by a gaze upward for two seconds. Ifthe system detects that pattern, then the Z-index for the relevant ARobject is adjusted. Thus, in embodiments, detecting an eye gaze patterncomprises detecting eyes focused in a first direction for apredetermined first time period, followed by detecting the eyes focusedin a second direction for a predetermined second time period.

FIG. 7 shows an eyewear apparatus 700 in accordance with embodiments ofthe present invention. Embodiments may include an eyewear apparatuscomprising a first lens configured and disposed to be worn in front of afirst eye of a user, a second lens configured and disposed to be worn infront of a second eye of a user, a first projection mechanism configuredand disposed to display the augmented reality object on the first lens,and a second projection mechanism configured and disposed to display theaugmented reality object on the second lens.

In the example, eyewear apparatus 700 comprises arms 704 and 706configured to rest on a user's ears, and lenses 710 and 712 configuredto be worn in front of the user's eyes. Eyewear apparatus 700 furthercomprises processing module 708 that contains some or all of the modulesdescribed with respect to device 100 in FIG. 1. Projection devices 714and 716 project AR images onto the lenses 710 and 712. Eyewear apparatus700 comprises a user facing camera 720 and a forward facing camera 718.Images 732 and 734 of the physical object 702 are seen through thelenses. AR objects 747 and 749 are seen projected onto the lenses 710and 712, respectively. Embodiments may utilize projectors, beamsplitters, waveguides, and/or other suitable technologies to renderimages on the lenses. In accordance with embodiments of the presentinvention, AR objects 747 and 749 may be rendered as “behind” all orpart of the respective images 732 and 734 of the physical object 702.

In embodiments, the user wears the eyewear apparatus 700, and moveshis/her gaze and/or head around as he/she looks around at thesurroundings. The dimensions of motion are labeled X, Y, and Z. The tiltpath, labeled T, is the amount the user tilts his/her head. The pandimension, labeled P, is the amount the user moves his/her head left andright.

In embodiments, as the user moves his/her head or his/her body, theprocessor (102 of FIG. 1) periodically recalculates an overlap region ofan AR object and the object it is obscuring. Based on a new set ofoverlap pixels, the opacity parameter (alpha) is adjusted accordingly.Pixels that were previously covering the first object but are no longercovering the object as a result of head motion have their correspondingopacity parameter reverted to its previous setting (e.g., opaque).Pixels that were previously not covering the first object but are nowcovering the object as a result of head motion have their correspondingopacity parameter set to a transparent (or partially transparent)setting to allow the first object that is “behind” the AR object tobecome visible.

In embodiments, the eyewear apparatus may utilize an optical combiner,such as a polarized beam combiner, and/or an off-axis combiner. Someembodiments may utilize a waveguide grating or waveguide hologram toextract a collimated image guided by total internal reflection (TIR) ina waveguide pipe. Any suitable projection and/or image combiningtechnique may be used.

FIG. 8 is a flowchart 800 indicating process steps for embodiments ofthe present invention. At 850, a first object is detected. At 852, anoverlapping augmented reality object is detected. The overlap may bepartial or complete. At 854, a user gaze is detected in the direction ofthe first object for a predetermined period of time. At 856, a Z-indexof the augmented reality object is adjusted. At 858, it is determinedwhether a user action is detected. The user action indicating that auser desires the Z-index adjusted is predetermined. If the user actionis detected, at 860, the Z-index is reverted. If the user action is notdetected, at 862, the Z-index is maintained.

Referring now to FIG. 9, a computerized implementation 900 of anembodiment for controlling a Z-index of an augmented reality object inan augmented reality system is described in further detail. Computerizedimplementation 900 is only one example of a suitable implementation andis not intended to suggest any limitation as to the scope of use orfunctionality of embodiments of the invention described herein.Regardless, computerized implementation 900 is capable of beingimplemented and/or performing any of the functionality set forthhereinabove.

In computerized implementation 900, there is a computer system 912.Examples of well-known computing systems, environments, and/orconfigurations that may be suitable for use with computer system 912include, but are not limited to, personal computer systems, servercomputer systems, thin clients, thick clients, hand-held or laptopdevices, multiprocessor systems, microprocessor-based systems, set topboxes, programmable consumer electronics, network PCs, minicomputersystems, mainframe computer systems, and distributed cloud computingenvironments that include any of the above systems or devices, and thelike.

This is intended to demonstrate, among other things, that the presentinvention could be implemented within a network environment (e.g., theInternet, a wide area network (WAN), a local area network (LAN), avirtual private network (VPN), etc.), a cloud computing environment, acellular network, or on a stand-alone computer system. Communicationthroughout the network can occur via any combination of various types ofcommunication links. For example, the communication links can compriseaddressable connections that may utilize any combination of wired and/orwireless transmission methods. Where communications occur via theInternet, connectivity could be provided by conventional TCP/IPsockets-based protocol, and an Internet service provider could be usedto establish connectivity to the Internet. Still yet, computer system912 is intended to demonstrate that some or all of the components ofimplementation 900 could be deployed, managed, serviced, etc., by aservice provider who offers to implement, deploy, and/or perform thefunctions of the present invention for others.

Computer system 912 is intended to represent any type of computer systemthat may be implemented in deploying/realizing the teachings recitedherein. Computer system 912 may be described in the general context ofcomputer system executable instructions, such as program modules, beingexecuted by a computer system. Generally, program modules may includeroutines, programs, objects, components, logic, data structures, and soon, that perform particular tasks or implement particular abstract datatypes. In this particular example, computer system 912 represents anillustrative system for gathering time-annotated web interaction andbiometric sensor data of web page users to extrapolate emotionalresponses to a web page and generating an aggregated emotional mapindicative of the emotional responses. It should be understood that anyother computers implemented under the present invention may havedifferent components/software, but can perform similar functions.

Computer system 912 in computerized implementation 900 is shown in theform of a general-purpose computing device. The components of computersystem 912 may include, but are not limited to, one or more processorsor processing units 916, a system memory 928, and a bus 918 that couplesvarious system components including system memory 928 to processor 916.

Bus 918 represents one or more of any of several types of busstructures, including a memory bus or memory controller, a peripheralbus, an accelerated graphics port, and a processor or local bus usingany of a variety of bus architectures. By way of example, and notlimitation, such architectures include Industry Standard Architecture(ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA)bus, Video Electronics Standards Association (VESA) local bus, andPeripheral Component Interconnects (PCI) bus.

Processing unit 916 refers, generally, to any apparatus that performslogic operations, computational tasks, control functions, etc. Aprocessor may include one or more subsystems, components, and/or otherprocessors. A processor will typically include various logic componentsthat operate using a clock signal to latch data, advance logic states,synchronize computations and logic operations, and/or provide othertiming functions. During operation, processing unit 916 collects androutes signals representing inputs and outputs between external devices914 and input devices (not shown). The signals can be transmitted over aLAN and/or a WAN (e.g., T1, T3, 56 kb, X.25), broadband connections(ISDN, Frame Relay, ATM), wireless links (802.11, Bluetooth, etc.), andso on. In some embodiments, the signals may be encrypted using, forexample, trusted key-pair encryption. Different systems may transmitinformation using different communication pathways, such as Ethernet orwireless networks, direct serial or parallel connections, USB,Firewire®, Bluetooth®, or other proprietary interfaces. (Firewire is aregistered trademark of Apple Computer, Inc. Bluetooth is a registeredtrademark of Bluetooth Special Interest Group (SIG)).

In general, processing unit 916 executes computer program code, such asprogram code for gathering time-annotated web interaction and biometricsensor data of web page users to extrapolate emotional responses to aweb page and generating an aggregated emotional map indicative of theemotional responses, which is stored in memory 928, storage system 934,and/or program/utility 940. While executing computer program code,processing unit 916 can read and/or write data to/from memory 928,storage system 934, and program/utility 940.

Computer system 912 typically includes a variety of computer systemreadable media. Such media may be any available media that is accessibleby computer system 912, and it includes both volatile and non-volatilemedia, removable and non-removable media.

System memory 928 can include computer system readable media in the formof volatile memory, such as random access memory (RAM) 930 and/or cachememory 932. Computer system 912 may further include otherremovable/non-removable, volatile/non-volatile computer system storagemedia, (e.g., VCRs, DVRs, RAID arrays, USB hard drives, optical diskrecorders, flash storage devices, and/or any other data processing andstorage elements for storing and/or processing data). By way of exampleonly, storage system 934 can be provided for reading from and writing toa non-removable, non-volatile magnetic media (not shown and typicallycalled a “hard drive”). Although not shown, a magnetic disk drive forreading from and writing to a removable, non-volatile magnetic disk(e.g., a “floppy disk”), and an optical disk drive for reading from orwriting to a removable, non-volatile optical disk such as a CD-ROM,DVD-ROM, or other optical media can be provided. In such instances, eachcan be connected to bus 918 by one or more data media interfaces. Aswill be further depicted and described below, memory 928 may include atleast one program product having a set (e.g., at least one) of programmodules that are configured to carry out the functions of embodiments ofthe invention. Program code embodied on a computer readable medium maybe transmitted using any appropriate medium including, but not limitedto, wireless, wireline, optical fiber cable, radio-frequency (RF), etc.,or any suitable combination of the foregoing.

Program/utility 940, having a set (at least one) of program modules 942,may be stored in memory 928 by way of example, and not limitation.Memory 928 may also have an operating system, one or more applicationprograms, other program modules, and program data. Each of the operatingsystem, one or more application programs, other program modules, andprogram data or some combination thereof, may include an implementationof a networking environment. Program modules 942 generally carry out thefunctions and/or methodologies of embodiments of the invention asdescribed herein.

Computer system 912 may also communicate with one or more externaldevices 914 such as a keyboard, a pointing device, a display 924, etc.;one or more devices that enable a consumer to interact with computersystem 912; and/or any devices (e.g., network card, modem, etc.) thatenable computer system 912 to communicate with one or more othercomputing devices. Such communication can occur via I/O interfaces 922.Still yet, computer system 912 can communicate with one or more networkssuch as a local area network (LAN), a general wide area network (WAN),and/or a public network (e.g., the Internet) via network adapter 920. Asdepicted, network adapter 920 communicates with the other components ofcomputer system 912 via bus 918. It should be understood that althoughnot shown, other hardware and/or software components could be used inconjunction with computer system 912. Examples include, but are notlimited to: microcode, device drivers, redundant processing units,external disk drive arrays, RAID systems, tape drives, and data archivalstorage systems, etc.

As can now be appreciated, disclosed embodiments provide techniques foradjusting the Z-index of an augmented reality (AR) object. Inembodiments, eye gaze and/or eye gestures can be used to change theZ-index of an AR object. In some embodiments, when an AR object covers afirst object, the user gazes at the first object for a predeterminedamount of time, and then the AR object Z-index is adjusted with respectto the Z-index of the first object, such that the AR object is nowrendered behind the first object, such that the user can continue toview the first object. This allows the user to conveniently continueviewing the first object. Embodiments provide techniques for revertingthe Z-index of the AR object at a later time when the user is ready torespond to the AR object. In this way, the user experience for augmentedreality systems is enhanced. Additionally, in some embodiments, the usergaze can focus on the AR object instead of the first object, in whichcase no Z-index adjustments are made.

Some of the functional components described in this specification havebeen labeled as systems or units in order to more particularly emphasizetheir implementation independence. For example, a system or unit may beimplemented as a hardware circuit comprising custom VLSI circuits orgate arrays, off-the-shelf semiconductors such as logic chips,transistors, or other discrete components. A system or unit may also beimplemented in programmable hardware devices such as field programmablegate arrays, programmable array logic, programmable logic devices, orthe like. A system or unit may also be implemented in software forexecution by various types of processors. A system or unit or componentof executable code may, for instance, comprise one or more physical orlogical blocks of computer instructions, which may, for instance, beorganized as an object, procedure, or function. Nevertheless, theexecutables of an identified system or unit need not be physicallylocated together, but may comprise disparate instructions stored indifferent locations which, when joined logically together, comprise thesystem or unit and achieve the stated purpose for the system or unit.

Further, a system or unit of executable code could be a singleinstruction, or many instructions, and may even be distributed overseveral different code segments, among different programs, and acrossseveral memory devices. Similarly, operational data may be identifiedand illustrated herein within modules, and may be embodied in anysuitable form and organized within any suitable type of data structure.The operational data may be collected as a single data set, or may bedistributed over different locations including over different storagedevices and disparate memory devices.

Furthermore, systems/units may also be implemented as a combination ofsoftware and one or more hardware devices. For instance, locationdetermination and alert message and/or coupon rendering may be embodiedin the combination of a software executable code stored on a memorymedium (e.g., memory storage device). In a further example, a system orunit may be the combination of a processor that operates on a set ofoperational data.

As noted above, some of the embodiments may be embodied in hardware. Thehardware may be referenced as a hardware element. In general, a hardwareelement may refer to any hardware structures arranged to perform certainoperations. In one embodiment, for example, the hardware elements mayinclude any analog or digital electrical or electronic elementsfabricated on a substrate. The fabrication may be performed usingsilicon-based integrated circuit (IC) techniques, such as complementarymetal oxide semiconductor (CMOS), bipolar, and bipolar CMOS (BiCMOS)techniques, for example. Examples of hardware elements may includeprocessors, microprocessors, circuits, circuit elements (e.g.,transistors, resistors, capacitors, inductors, and so forth), integratedcircuits, application specific integrated circuits (ASIC), programmablelogic devices (PLD), digital signal processors (DSP), field programmablegate array (FPGA), logic gates, registers, semiconductor devices, chips,microchips, chip sets, and so forth. However, the embodiments are notlimited in this context.

Also noted above, some embodiments may be embodied in software. Thesoftware may be referenced as a software element. In general, a softwareelement may refer to any software structures arranged to perform certainoperations. In one embodiment, for example, the software elements mayinclude program instructions and/or data adapted for execution by ahardware element, such as a processor. Program instructions may includean organized list of commands comprising words, values, or symbolsarranged in a predetermined syntax that, when executed, may cause aprocessor to perform a corresponding set of operations.

The present invention may be a system, a method, and/or a computerprogram product at any possible technical detail level of integration.The computer program product may include a computer readable storagemedium (or media) having computer readable program instructions thereonfor causing a processor to carry out aspects of the present invention.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, may be non-transitory,and thus is not to be construed as being transitory signals per se, suchas radio waves or other freely propagating electromagnetic waves,electromagnetic waves propagating through a waveguide or othertransmission media (e.g., light pulses passing through a fiber-opticcable), or electrical signals transmitted through a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device. Program data may also bereceived via the network adapter or network interface.

Computer readable program instructions for carrying out operations ofembodiments of the present invention may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, or either source code or object code written in anycombination of one or more programming languages, including an objectoriented programming language such as Smalltalk, C++ or the like, andconventional procedural programming languages, such as the “C”programming language or similar programming languages. The computerreadable program instructions may execute entirely on the user'scomputer, partly on the user's computer, as a stand-alone softwarepackage, partly on the user's computer and partly on a remote computer,or entirely on the remote computer or server. In the latter scenario,the remote computer may be connected to the user's computer through anytype of network, including a local area network (LAN) or a wide areanetwork (WAN), or the connection may be made to an external computer(for example, through the Internet using an Internet Service Provider).In some embodiments, electronic circuitry including, for example,programmable logic circuitry, field-programmable gate arrays (FPGA), orprogrammable logic arrays (PLA) may execute the computer readableprogram instructions by utilizing state information of the computerreadable program instructions to personalize the electronic circuitry,in order to perform aspects of embodiments of the present invention.

These computer readable program instructions may be provided to aprocessor of a computer, or other programmable data processing apparatusto produce a machine, such that the instructions, which execute via theprocessor of the computer or other programmable data processingapparatus, create means for implementing the functions/acts specified inthe flowchart and/or block diagram block or blocks. These computerreadable program instructions may also be stored in a computer readablestorage medium that can direct a computer, a programmable dataprocessing apparatus, and/or other devices to function in a particularmanner, such that the computer readable storage medium havinginstructions stored therein comprises an article of manufactureincluding instructions which implement aspects of the function/actspecified in the flowchart and/or block diagram block or blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

While the disclosure outlines exemplary embodiments, it will beappreciated that variations and modifications will occur to thoseskilled in the art. For example, although the illustrative embodimentsare described herein as a series of acts or events, it will beappreciated that the present invention is not limited by the illustratedordering of such acts or events unless specifically stated. Some actsmay occur in different orders and/or concurrently with other acts orevents apart from those illustrated and/or described herein, inaccordance with the invention. In addition, not all illustrated stepsmay be required to implement a methodology in accordance withembodiments of the present invention. Furthermore, the methods accordingto embodiments of the present invention may be implemented inassociation with the formation and/or processing of structuresillustrated and described herein as well as in association with otherstructures not illustrated. Moreover, in particular regard to thevarious functions performed by the above described components(assemblies, devices, circuits, etc.), the terms used to describe suchcomponents are intended to correspond, unless otherwise indicated, toany component which performs the specified function of the describedcomponent (i.e., that is functionally equivalent), even though notstructurally equivalent to the disclosed structure which performs thefunction in the herein illustrated exemplary embodiments of theinvention. In addition, while a particular feature of embodiments of theinvention may have been disclosed with respect to only one of severalembodiments, such feature may be combined with one or more features ofthe other embodiments as may be desired and advantageous for any givenor particular application. Therefore, it is to be understood that theappended claims are intended to cover all such modifications and changesthat fall within the true spirit of embodiments of the invention.

What is claimed is:
 1. A computer-implemented method comprising:detecting a first object as currently focused on by a user; detectingpresence of an overlapping augmented reality object that obstructs aview of the first object; determining that the user is maintaining gazeon the first object for at least a predetermined threshold period oftime; rendering a confirmation message; wherein the confirmation messagecomprises a first area and a second area, wherein the first areacomprises an indication that adjustment is desired, and wherein thesecond area comprises an indication that adjustment is not desired;detecting the user gaze on a first area of the confirmation message, andresponsive to the determination and the detection of the user gaze onthe first area of the confirmation message, adjusting an opacityparameter of the augmented reality object.
 2. The computer-implementedmethod of claim 1, wherein the adjusting the opacity parameter comprisesreverting the Z-index for the augmented reality object.
 3. Thecomputer-implemented method of claim 1, wherein the overlappingaugmented reality object only obstructs a partial view of the firstobject.
 4. The computer-implemented method of claim 3, wherein theadjusting the opacity parameter is performed only for the obstructingportion of the augmented reality object.
 5. The computer-implementedmethod of claim 1, further comprising moving a position of theoverlapping augmented reality object to a non-overlapping location withrespect to the first object.
 6. The computer-implemented method of claim1, wherein the first object is a physical object.
 7. Thecomputer-implemented method of claim 1, wherein the first object is avirtual reality object.
 8. An electronic communication devicecomprising: a forward-facing digital camera; a user-facing digitalcamera; a processor; a memory coupled to the processor, the memorycontaining instructions, that when executed by the processor, performthe steps of: detecting a first object as currently focused on by auser; detecting presence of an overlapping augmented reality object thatobstructs a view of the first object; and determining that the user ismaintaining gaze on the first object for at least a predeterminedthreshold period of time; rendering a confirmation message; wherein theconfirmation message comprises a first area and a second area, whereinthe first area comprises an indication that adjustment is desired, andwherein the second area comprises an indication that adjustment is notdesired; detecting the user gaze on a first area of the confirmationmessage, and responsive to the determination and the detection of theuser gaze on the first area of the confirmation message, adjusting anopacity parameter of the augmented reality object.
 9. The electroniccommunication device of claim 8, wherein the adjusting the opacityparameter comprises changing the Z-index for the augmented realityobject.
 10. The electronic communication device of claim 8, wherein theoverlapping augmented reality object only obstructs a partial view ofthe first object.
 11. The electronic communication device of claim 10,wherein the adjusting the opacity parameter is performed only for theobstructing portion of the augmented reality object.
 12. The electroniccommunication device of claim 8, wherein the memory further comprisesinstructions, that when executed by the processor, perform the step of,moving a position of the overlapping augmented reality object to anon-overlapping location with respect to the first object.
 13. Theelectronic communication device of claim 8, wherein the first object isa physical object or a virtual reality object.
 14. The device of claim8, further comprising an eyewear apparatus comprising: a first lensconfigured and disposed to be worn in front of a first eye of a user; asecond lens configured and disposed to be worn in front of a second eyeof a user; a first projection mechanism configured and disposed todisplay the augmented reality object on the first lens; and a secondprojection mechanism configured and disposed to display the augmentedreality object on the second lens.
 15. A computer program productcomprising a computer readable storage medium having programinstructions embodied therewith, the program instructions executable bya processor to cause the electronic computing device to: detect a firstobject as currently focused on by a user; detect presence of anoverlapping augmented reality object that obstructs a view of the firstobject; determine that the user is maintaining gaze on the first objectfor at least a predetermined threshold period of time; render aconfirmation message; wherein the confirmation message comprises a firstarea and a second area, wherein the first area comprises an indicationthat adjustment is desired, and wherein the second area comprises anindication that adjustment is not desired; detect the user gaze on afirst area of the confirmation message, and responsive to thedetermination and the detection of the user gaze on the first area ofthe confirmation message, adjust an opacity parameter of the augmentedreality object.
 16. The computer program product of claim 15, whereinthe adjusting the opacity parameter comprises changing the Z-index forthe augmented reality object.
 17. The computer program product of claim15, wherein the overlapping augmented reality object only obstructs apartial view of the first object.
 18. The computer program product ofclaim 17, wherein the adjusting the opacity parameter is performed onlyfor the obstructing portion of the augmented reality object.
 19. Thecomputer program product of claim 15, further comprising moving aposition of the overlapping augmented reality object to anon-overlapping location with respect to the first object.
 20. Thecomputer program product of claim 15, wherein the first object is aphysical object or a virtual reality object.